


Feedback Rates on FFN and AO3

by longlivefeedback



Category: No Fandom
Genre: LLF Comment Project, Other, data analysis, feedback
Language: English
Status: In-Progress
Published: 2018-02-24
Updated: 2018-03-20
Packaged: 2019-03-23 11:54:33
Rating: General Audiences
Warnings: No Archive Warnings Apply
Chapters: 4
Words: 2,811
Publisher: archiveofourown.org
Story URL: https://archiveofourown.org/works/13787151
Author URL: https://archiveofourown.org/users/longlivefeedback/pseuds/longlivefeedback
Summary: Analysis and discussion of written feedback frequency (comments and reviews) on fanfiction.net and archiveofourown.org.





	1. Introduction

##  **Introduction**

At the beginning of January 2018, we set out to [collect some data](https://longlivefeedback.tumblr.com/post/169468381258/feedback-study) on commenting rates on two major fanfiction sites, fanfiction.net (FFN) and archiveofourown.org (AO3), with the goal of determining whether stories receive more feedback on one site or the other.

For those unfamiliar with either platform, FFN and AO3 allow writers to post their works and receive feedback from readers. Readers can leave written feedback at the end of each chapter of the fic, which can then be displayed publicly. Written feedback is referred to as “reviews” on FFN and “comments” on AO3.

For this study, we asked respondents to submit four data points corresponding to each of their stories: the review count, the number of views on FFN, the number of comment threads on AO3, and the number of hits on ao3. To be eligible, stories must have only one chapter and be posted on both sites.

 

##  **Assumptions and Limitations**

  * Results only apply to single-chapter stories - we don’t know if or how this information relates to multi-chapter fics.
  * Samples may not accurately represent AO3 and FFN as a whole. The goal of this study is to provide a general overview, and therefore the reality may vary between fandoms and time periods.
    * Sampling bias: respondents are likely to be AO3-dominant users, and thus spend more time building community on AO3, which could increase the feedback rate.
    * Over- and under-representation of fandoms: if the feedback rate varies between sites by fandom, this could throw off results.
    * Does not control for date posted, and therefore may be affected if feedback rate has changed significantly over time and/or fics were posted on each site on different times. However, because we are measuring feedback rate instead of total feedback, and have a large sample size, we don’t expect this to affect the results.
  * Hits are counted differently on each website - however, this is ameliorated by only including single-chapter stories.
    * FFN counts each visit as a hit for the overall count, and tracks visitors separately. FFN hit trackers indicate that hits do approximate the number of visitors for one-chapter stories, but cannot be used to determine the exact number of readers.
    * AO3 counts a new hit “when a user visits a work and another user has visited that work since the last time they viewed it.”
  * Guest comments can be disabled on AO3, while guest reviews can be moderated but not disabled on FFN. This is likely to decrease the overall feedback frequency on ao3, but we do not know if the effect is significant.
  * These data do not reflect the length or intricacy of feedback - a one word review is counted the same as a five hundred word comment, and vice versa.
  * We are unable to account for bot activity - aka, webscrapers, search engine bots, etc. 




	2. Characterization of Data

**Summary for the Chapter:**

> Overview of hits, comments/reviews, and feedback frequency.

##  **Who responded?**

Using tumblr’s reblog graph feature, we were able to determine which fandoms are likely to make up the majority of the data by analyzing the audience of the [original LLF post](https://longlivefeedback.tumblr.com/post/169313240823/feedback-rates-ao3-vs-ffn) containing the survey link and explanation. From this, we can assume that some fandoms are underrepresented in terms of their overall popularity (Voltron), and others are statistically overrepresented (Tolkien). We also know that we have collected data from multiple fandoms, and so feedback rates in one particular overrepresented fandom are less likely to throw off our results as a whole.

_**Figure 1.** Graph of influential reblogs of the original LLF tumblr post containing the survey link and explanation. _

##  **Reviews and comments and hits, oh my!**

The survey received a total of 522 responses - aka, 522 fics.

On FFN, the mean number of views per fic is approximately 1,008, and the mean number of reviews is 5.01. The middle 50% have 133-850 views and 1-6 reviews. Fics with at least one review (444 out of 521 valid responses) have an average of 10.56 reviews per thousand hits, and the middle 50% have 4-13.46 reviews per thousand hits. The maximum number of views is 22,445 and the maximum number of reviews is 100.

On AO3, the mean number of hits is 1,339.5. The mean number of comments is 4.72. Half the stories have 257-1354 hits and 1-6 comments. Fics with at least one comment (429 out of 522) have an average of 8.67 comments per thousand hits, and the middle 50% have 2.36-11.21 comments per thousand hits. The maximum number of hits is 27,315 and the maximum number of comments is 76.

Please note that these calculations include all outliers.

_**Table 1.** Statistics for all viable (ie, without obvious entry errors) fics as summarized above. _

__

_**Table 2.** Stats for all viable fics with at least one comment or one review, as summarized above. _

__

_**Figure 2.** Box and whiskers plot of views on FFN (blue) and AO3 (red) with outliers (1.5 IQR of full data set below Q1 or above Q3) removed. FFN includes 462 fics, and AO3 includes 475.  _

__

_**Figure 3.** Box and whiskers plot of feedback on FFN (blue) and AO3 (red) with outliers removed - note that this shows AO3 with a greater mean number of comments than FFN. FFN includes 472 fics, while AO3 includes 477. _

##  **Feedback Rates**

This is a frequency histogram bucketed by feedback rate (the number of comments or reviews divided by the number of hits). The data are heavily skewed right, which means that the majority of stories are clustered around lower feedback rates.

_**Figure 4.** Frequency histogram comparing FFN (blue) and AO3 (red) feedback rates. _

__

_**Figure 5.**  Proportion of fics that have a higher feedback rate on FFN (blue, 71%) and AO3 (red, 29%), out of stories that have at least one review/comment on both sites (375 out of 521 fics). _

##  **Next up: is this even a thing?**

The initial characterization of our data indicates that FFN does tend to have a higher feedback rate than AO3. So, is that the end of it?

Well, no!

What we have right now are just numbers - we can’t say whether there’s an actual difference, or if these results might just be due to chance. For example, are fics that have more reviews on FFN overrepresented in our data? We also haven’t delved into the outliers, and while the overall metrics indicate that FFN has a higher feedback rate, AO3 has a higher mean and median number of comments once outliers are removed ( _see_  Figure 3). Therefore, in the next installment, we will be conducting significance testing to determine whether or not the observed difference is really a difference, getting into detail work, and discussing the implications of our results.


	3. Significance Testing

**Summary for the Chapter:**

> Determining statistical significance of differences in feedback frequency.

## Results

The data we collected include a total of 521 viable fics on FFN and 522 viable fics on AO3. FFN results show means of 1,008.2 views, 5.01 reviews per fic, and 9.00 reviews for every thousand views. AO3 results show means of 1,339.5 hits, 4.72 comment threads per fic, and 7.13 comment threads for every thousand hits.

Overall, our data show that FFN has fewer views, but more reviews per fic and per thousand hits than AO3.

In order to determine the statistical significance of the differences, we subtracted the AO3 feedback rate (comments divided by hits) from the FFN feedback rate. Outliers (1.5 IQR above Q3 or below Q1) were removed, resulting in a sample of 419 values. We confirmed that the resulting distribution was approximately normal. Then, we performed a t-test to show whether the resulting mean of 0.002 (ie, FFN has a mean of 2 reviews per thousand hits than AO3) was significant.

**Figure 1.**   _Frequency distribution of AO3 feedback rate (comments/hits) subtracted from FFN feedback rate (reviews/views), with outliers 1.5 IQR above Q3 or below Q1 removed._

Using H0 = 0, n= 419, s.d. = 0.00463463 we get a t-stat of 8.987783 which translates into a p-value of 8.790e-18. In other words, p < 0.001, which is well below the generally accepted threshold of p = 0.05 (a 95% probability that results are not due to random chance). Therefore, we conclude that there is a statistically significant difference between the feedback rates on FFN and AO3 as measured by comments/hits.

When data are normally distributed, approximately 68% of differences will fall within one standard deviation of the mean (0.002 ∓ 0.0046), 95% will fall within two standard deviations (0.002 ∓ 0.0093), and 99.7% will fall within three standard deviations (0.002 ∓ 0.0139). However, due to the fact that removing outliers took out nearly 20% of our data pairs, we will need to explore this further to determine whether this accurately describes fics - a brief glance suggests that outliers primarily consist of stories that have zero comments or reviews on one site and a nonzero amount of feedback on the other site, indicating that these results do apply to fics that have at least one comment or review on both sites.

 

We also ran our data through the [Wilcoxon signed-rank](https://t.umblr.com/redirect?z=https%3A%2F%2Fen.wikipedia.org%2Fwiki%2FWilcoxon_signed-rank_test&t=YTkwZjQwOWE4NWY2ZDRiMjZmY2VhN2U1NWZhYzZmMWFlYjc1YjIxZCxXUXoyZFlkYg%3D%3D&b=t%3AIpAB_w2_aqVUbdtX6pX9WA&p=https%3A%2F%2Flonglivefeedback.tumblr.com%2Fpost%2F171039248108%2Ffeedback-rates-on-ffn-and-ao3-significance&m=0) test. Following [this methodology](https://t.umblr.com/redirect?z=http%3A%2F%2Fwww.real-statistics.com%2Fnon-parametric-tests%2Fwilcoxon-signed-ranks-test%2F&t=YjM4NzYyMjQ1ODIwNGY0OTE1MzhjNjk1OTgxNDBjMmJkNzc3MjU3NixXUXoyZFlkYg%3D%3D&b=t%3AIpAB_w2_aqVUbdtX6pX9WA&p=https%3A%2F%2Flonglivefeedback.tumblr.com%2Fpost%2F171039248108%2Ffeedback-rates-on-ffn-and-ao3-significance&m=0), we got n =  497, T = 41,964.5, mean = 61,876.5, and var = 10,261,186.3 resulting in a Z-score of 6.2161 with a p-value of 5.0975e-10. Once again, we find that these results are highly significant.

## Relationship Between Feedback Rates

Correlation between FFN reviews-to-views ratio and AO3 comment threads-to-hits rate is 0.2567 for all viable fics (n = 521), which means that there is not a strong relationship between a fic’s feedback rate on AO3 and its feedback rate on FFN. This indicates that a story’s level of popularity and audience engagement is not determined solely by the fic itself, but is instead influenced by a number of other factors.

**Figure 2.** _Graph showing the relationship between the FFN reviews-to-views ratio and the AO3 reviews-to-views ratio of the 521 viable data pairs. Not all data points are shown._

## Conclusions

Our results indicate that the difference in feedback rates (in this case, comments or reviews per hit) between FFN and AO3 are highly significant (p < 0.001). We also found that FFN has a mean of 2 additional reviews per thousand hits as compared to the same fic on AO3. Note that this does not indicate that every fic will receive a higher feedback rate on FFN: in our initial characterization, we found that out of stories that have at least one review or comment on each site, 29% have a higher feedback rate on AO3. Furthermore, we have shown that there is not a strong correlation between the feedback rates on both sites. This indicates that the fic itself is only one factor determining popularity and audience engagement.

We caution that these results must be interpreted within the context of our assumptions and limitations - that is, we cannot say whether or not they apply to fics as a whole. Our sample is presumably biased towards authors who primarily use AO3, and thus may have a higher feedback rate as discussed in our initial data characterization. Additionally, we only looked at single-chapter works, and it is unclear if, or how, these results may apply to multichapter fics.

We do not intend to discourage any authors from posting on AO3, but instead investigate feedback rates with the goal of using this information to determine what may be responsible for the differences, and whether we can use our findings to propose changes to increase feedback rates on AO3. Furthermore, this only takes into account one form of feedback, and neglects others - most significantly, AO3’s kudos feature. In fact, if any conclusions about which platform to use can be drawn from these data, our suggestion is… both. If stories are permitted on both sites, as FFN does not allow explicit fics, we recommend that authors crosspost their works.

In our next section, we intend to discuss the potential causes for FFN’s higher feedback rate based on what we know about how readers interact with fics and commenting, why people choose to comment (or not to comment), and, in particular, what role we believe kudos might play.


	4. Feedback Rate and Hits

**Summary for the Chapter:**

> Fics with more hits have a lower mean feedback frequency on both FFN and AO3.

**Introduction**

  
This is part of LLF’s [ongoing project](https://longlivefeedback.tumblr.com/post/169468381258/feedback-study) analyzing feedback frequencies on FFN and AO3. In this section, we will be looking at the differences between feedback rates based on the number of hits. After our previous tests, we became interested in how traffic levels affect the feedback rate after observing that fics with fewer than a few hundred hits are more likely to have at least one comment than indicated by our overall averages. We hypothesized that, too, that readers are less likely to comment on a story if they see that it already has more than a handful of comments, and perhaps are more likely to comment if it has no comments.

 

**Results**

  
In order to do this, we separated the stories into four “traffic” categories based on the quartiles found in our [ initial characterization ](https://longlivefeedback.tumblr.com/post/170085003848/feedback-rates-on-ffn-and-ao3-initial).

_Table 1._ Traffic categories.

 

_Figure 1._ Frequency histogram showing feedback rates, as calculated by the number of comment threads divided by the number of hits, for AO3 fics by traffic category. Not all data is shown (cut off at 0.03 comments per hit, or 30 comments per 1000 hits).

_Figure 2._ Frequency histogram showing feedback rates, as calculated by the number of reviews divided by the number of views, for FFN fics by traffic category. Not all data is shown (cut off at 0.03 reviews per hit, or 30 reviews per 1000 hits).

 

As expected, low-traffic fics are more likely to have zero comments (54 out of 132 fics on AO3, and 61 out of 130 fics on FFN). We can also see indications sharp decline in the feedback rate for the other traffic categories - i.e., a cursory glance indicates that a higher hitcount is associated with a lower feedback frequency.

However, does the mean feedback rate per category reflect this trend, when zero-comment fics are included in the calculations?

Spoiler alert: yes.

_Figure 3._ Average feedback rate by traffic category of fics on AO3. 

_Figure 4._ Average feedback rate by traffic category of fics on FFN.

 

By now, all you stats-minded folk are rubbing your hands together and demanding significance tests. Fear not!

We ran a Kruskal-Wallis test on the AO3 and FFN samples. Both showed highly significant differences at p < 0.01. To confirm the difference between the low and mid-low categories on FFN, we also performed an individual Mann-Whitney U test (p < 0.05).   


**Conclusions**

We found that stories with more views have a significantly lower mean feedback rate. While this is in some senses intuitive - readers may feel that there’s no need to comment on a fic that already has a significant number of comments - it is also surprising the increased popularity of a story decreases the feedback rate. This is supported by our findings in our tests of the difference in feedback rates between AO3 and FFN, which showed that the same fic was likely to have very different feedback rates between the two sites. Therefore, we conclude that the story itself is just one part of what determines reader engagement through written feedback.

We also observe that the difference between traffic categories seems to be larger on AO3 than it is on FFN. This may be due to the size of the categories and data spread - AO3’s low traffic category includes stories with up to 257 hits, while FFN’s cutoff is 133 views. However, if readers are indeed less likely to comment on a story if they perceive it as already having many comments, this may be exacerbated by the site format. On AO3, the total number of comments per fic and chapter, including author replies, is prominently displayed at both the top and above the comment section. However, FFN only shows the number of reviews at the top of the story and _not_ above the review box. Additionally, replies are conducted entirely through the private messaging system and so do not count towards the total.

Therefore, on AO3, readers will always see the number of comments, while on FFN, they frequently will not know how many other reviews have been left without looking back at the top of the fic. On high-traffic fics, this may contribute to a higher feedback rate, because readers are not discouraged or dissuaded by the number of reviews - however, if readers are more likely to review zero-comment fics, they may be less likely to leave feedback if they cannot see that the story does not have any comments.

At this point, this is all conjecture - our theories as to why we’re seeing what we’re seeing are merely ideas we have and not explanations supported by concrete proof. As we continue to collect and analyze data, we hope to be able to shed more light on this topic!

**Author's Note:**

> This work is part of [LLF Comment Project](https://longlivefeedback.tumblr.com/llfcommentproject), whose goal is to improve communication between readers and authors. This author invites:
> 
>   * Short comments
>   * Long comments
>   * Questions and clarification requests
>   * Constructive criticism
>   * Corrections
> 

> 
> While this is a study as opposed to a story, we greatly value all feedback and support. We will reply to all comments as time allows, prioritizing corrections and questions.


End file.
